DOCUMENT RESUME 



ED 464 146 



TM 033 830 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



Hoban, Gary; Neu, Beverly; Castle, Sidney R. 

Assessment of Student Learning in an Educational 
Administration Online Program. 

2002-04-00 

28p . ; Paper presented at the Annual Meeting of the American 
Educational Research Association (New Orleans, LA, April 
1-5, 2002) . Colored figures may not reproduce well. 
Numerical/Quantitative Data (110) -- Reports - Research 

(143) -- Speeches/Meeting Papers (150) 

MF01/PC02 Plus Postage. 

♦Academic Achievement; *College Faculty; Distance Education; 
♦Educational Administration; Educational Quality; *Graduate 
Students; Graduate Study; Higher Education; *0nline Courses; 
Student Attitudes; *Student Evaluation; Student Surveys; 
Teacher Attitudes 



ABSTRACT 



This study was conducted to determine how to know what 
online students learned in an educational administration program and whether 
it was comparable to what students learned in traditional (on ground) 
classes. The focus was on the educational administration program at National 
University, California. Surveys of 21, 47, and 11 online students were 
completed to measure consistency of student attitude toward online learning 
in relation to traditional courses. Scores on a comprehensive exit 
examination were compared for on-ground (n=125) and online (n=ll) students. 
Students were also asked to assess their own learning in both conditions. 

Nine online instructors completed a survey that gave their perspectives of 
the online courses and their effects on students. Results suggest a high 
degree of comparability in the quality and rigor of the online and on-ground 
educational administration courses. Online students were generally satisfied 
with the quality of instruction in their program, Students in both types of 
courses performed comparably academically. Online instructors were quite 
positive about the educational administration program. Overall, findings show 
that students enrolled in online courses, even if they have concerns, like 
the flexibility the course gives them and plan to take more courses in this 
form. (Contains 11 tables, 8 graphs, and 15 references.) (SLD) 
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ASSESSMENT OF STUDENT LEARNING IN AN 
EDUCATIONAL ADMINISTRATION ONLINE PROGRAM 



Gary Hoban, Ph.D., National University 
Beverly Neu, Ph.D., University of Southern California 
Sidney R. Castle, Ph.D., National University 



Introduction 

The traditional method of preparing students for a credential or certificate authorizing 
service as a school administrator has been through a sequence of on-ground classes. 
Students and professors meet in a standard face-to-face environment for the number of 
contact hours prescribed by the appropriate accrediting agency. Through external 
accreditation review, university assessments of student outcomes, and student reports of 
self-learning, it is generally accepted that student learning in these programs can be 
demonstrated and further validated by research. 

Today, a number of educational administration students are pursuing their credential or 
certificates online. It is estimated that 2.2 million people will have taken an online 
college or university course by 2002 ( San Francisco Chronicle, August 14, 2000). At 
National University, a university which primarily serves the working adult learner — 
whose average age is thirty-four — more and more students are preparing to become 
school administrators through online courses and programs. Last year (2000) twenty 
students completed all or part of their educational administration program online. This 
year (2001-2002) approximately seventy students are using the online option. The 
number fluctuates from course to course, but is fairly consistent throughout the program. 

Colleges and universities throughout the world offer online programs. It is virtually 
impossible to find official data on the number of on lin e college and university courses 
currently being offered. Attempts to determine numbers through Internet searching prove 
frustrating. A recent search using the Lycos search engine with the term “online 
university education” as the descriptor indicated that 1,467,325 articles on the topic were 
available. Clearly online instruction is no longer the domain of non-traditional 
universities with schools such as Harvard now offering an online MBA. 

Indicative of this trend, eleven universities in the United States and Britain have joined 
together to create and sell online graduate courses, thus forming one of the more 
extensive international distance-learning partnership, ( Chronicle of Higher Education, 
June 8, 2001). The network includes Pennsylvania State University, the University of 
California at San Diego, the University of Illinois at Urbana-Champaign, the University 
of Washington, and the University of Wisconsin at Madison. The British partners are the 
Universities of Bristol, Leeds, Manchester, Sheffield, Southampton, and York. The 
initial focus is on research collaborations. 
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Background: National University 

National University is one of the pioneers in offering online instruction, having begun in 
1996 by offering a Global Masters in Business Administration. Since then, it has offered 
degree and credential programs in eighteen different fields: BA — Global Studies, 

Business Administration, Criminal Justice, and Nursing: masters — Business 
Administration, Electronic Commerce, Forensic Science, Nursing, Instructional 
Technology, Educational Technology, Teaching, and Educational Administration; 
Credentials — Multiple Subject, Single Subject (CLAD and BCLAD), and Educational 
Administration; and Certificates — CLAD Multiple and Single Subject and Electronic 
Commerce. Additional programs will be offered online within the next year. Nearly ten 
percent of National University’s 18,000 students complete all or part of their programs 
online. 

The University of Southern California has been involved with online instruction since 
1 997. Its major program is the Master of Gerontology offered by the Leonard Davis 
School of Gerontology and the online journalism programs offered through the 
Annenburg Institute. 

The focus of this paper and presentation, however, is on the educational administration 
program at National University where Hoban and Castle teach as full-time professors and 
Neu teaches as an adjunct professor while coordinating a Tier 2 (Professional— advanced 
credential) educational administration program at the University of Southern California. 

Focus of This Study — Educational Administration Online at National University 

The main reason students give for taking online educational administration courses is 
convenience and flexibility of time. Those students who participated in the first National 
University offering of the preliminary administrative services credential program in 2000 
reported that observation and noted that the online courses were as rigorous and 
demanding as those they have taken in the more traditional format. This was confirmed 
in a preliminary study conducted by Hoban and Neu, “What We Know and What we 
Want to Know about Online Education" which was presented at the 2001 AERA 
Conference in Seattle. 

Online course work, distance education, not only allegedly saves costs, but it also reaches 
a wider student audience and better addresses the needs of students. More importantly, it 
applies the tenets of modem learning pedagogy (Fitzpatrick, Robbie, 2001). Preliminary 
evidence from the Hoban and Neu study also indicated that this is the case for the 
National University program. 

A study by L. Sherry, 1996, found that political and public interest in distance education 
is especially high in regions where the student population is widely distributed. Sherry 
also found that each region developed its own form of distance education in accordance 
with local resources, target audience, and the philosophy of the organizations that provide 
the instruction. Currently, the National University program has cohorts of students from 
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throughout California with several students residing in other “lower 48” states and in 
Mexico. The program also has two special cohorts of students enrolled in Alaska. 

New developments in technology have removed some of the early disadvantages 
associated with distance education. In 1 984, a long time ago in the history of distance 
education. Bates suggested that new technologies promise “a wider range of teaching 
functions and a higher quality of learning, lower costs, greater student control, more 
interaction and feedback for students” (223). Today that appears to be the case. And, the 
old issue of media vs. method will continue to be debated in relation to distance 
education. There is no doubt that distance education is different from other instructional 
approaches (Jeffries, retrieved 2002). 

National University Online 

National University has been preparing students in educational administration leading to 
the California Preliminary Administrative Services Certificate (Tier I) using a traditional, 
face-to-face on-ground format for twenty years. This certificate is required to be a 
principal as well as to seek other administrative positions in the State of California. 

National University developed an online program leading to the Preliminary 
Administrative Services Certificate for the 2000-2001 academic year. The first courses 
in the program were taught that year. The experiences of several of the instructors were 
described in a paper, “Online Education — Notes From the Trenches — Purebred vs. 
Hybrid,” presented at the XVIII International Council for Innovation in Higher Education 
(ICIE) conference held in Quebec City, in November, 2000. Twenty students have 
completed or are in the final stages of completing the program online and 70-80 students 
are currently enrolled in parallel online programs for the 2001-2002 academic year. Two 
groups of students, as noted above, reside in Alaska. 

An initial formal assessment of the program was conducted and reported in a paper, 
“Adult Learners in 2001” that was presented at the ICIE conference held in Rome, Italy, 
in November, 2001 . That assessment primarily focused on a second group of students’ 
attitudes toward online learning and was compared to the information reported from the 
first survey presented by Hoban and Neu at the Seattle 2001 AERA meeting. Of 
particular interest were the student comments which, for the most part, gave a very strong 
endorsement to the online approach. 

While the students responding to the first two surveys appeared to be quite satisfied with 
their online educational administration programs, it is still relatively early to reach any 
definitive conclusions about actual student learning in this environment. The content of 
on-ground and online classes is the same, as are textbooks, assignments, and readings. 
Replacing face-to-face discussions are online threaded discussions and, at program 
intervals, mandated face-to-face classes. Anecdotal evidence strongly suggests that 
online students have learned as much as non-online students. So, too, does student 
performance on program exit examinations and field-work portfolios. That evidence, 
however, is limited. As a consequence, the question which must be explored by online 
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educational administration instructors at National University and , at other universities, 
too, is, “How do we know what the online students have learned and is it comparable to 
what students have learned in on-ground classes? ” 



The Questions To Be Asked 

To answer the central question posed in this paper — How do we know what the online 
students have learned and is it comparable to what students have learned in on-ground 
classes? — the following sub-questions are addressed in this study. 

1. Is there a consistency of student attitude response regarding student perceptions of the 
quality of instruction they have received in the online format? This question is addressed 
through summarizing the results of three student attitude surveys — two of which were 
noted above — which look at 12 dimensions of online instruction in the educational 
administration program at National University. The first survey gathered information 
from students in the first offering of the program in 2000, the second in the second 
offering in 2001, and the third from a group of students completing the program in 
Alaska in 2001-2002. In addition, information from student assessments of their own 
learning based upon the responses to nine attitude related issues on the University’s 
student evaluation forms — for all online students and a comparable sampling of on- 
ground students for the time period in question — is evaluated. 



2. Is the performance of students in the online program comparable to that of students in 
the on-ground program? Currently, the National University educational administration 
program assessment process utilizes several measures — student surveys, grades, and a 
comprehensive exit examination. The comprehensive exit examination measures student 
knowledge in seven content domains directly related to the established performance 
outcomes for the program which are prescribed by the state. The results of online 
students who have completed the program are compared to the mean results of students 
who have completed the program on-ground in 2000-2001. The “n” is quite small and 
thus the results can only be seen as an indicator of comparability at this time. 

3. How do students assess their learning in online instruction in comparison to students 
who have completed the same courses on-ground. The student course evaluation forms — 
for both online and on-ground courses — used at National University pose three 
summative questions: assessment of the instructor, assessment of the quality of 
instruction, and student assessment of their own learning. These questions are answered 
using a five-point scale with one being low and five being high. The mean scores on each 
item for each online course completed in the educational administration program and an 
equal number of comparable on-ground courses are examined with appropriate analyses 
being made. 

4. How do the instructors of the on-line courses in the educational administration 
program at National University evaluate the academic performance of their online 
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students in relationship to the performance of students they have had in comparable on- 
ground courses. An online instructor survey has been administered to the instructors in 
the program and the results are reported. 

The information gathered in response to these four sub-questions is used to address the 
overall question of the study. Because the information comes from a variety of data 
sources, it is not offered as definitive but as illustrative of trends and, it is hoped, sheds 
light on how well educational administration students in the online program at National 
University are performing. 

Consistency of Student Attitudes Toward Online Education 

Survey of Online Learners 

In 2000, during the first offering of the educational administration sequence, 21 students 
were surveyed regarding their attitudes toward online learning in relationship to their 
experiences as on-ground students in other programs. In 2001, 47 new students in the 
second offering of the program were likewise surveyed and then, later in that year, 1 1 
additional students from a cohort of students enrolled in an Alaska also responded to a 
separate request for feedback. All of the surveys were completed anonymously and 
participation was strictly voluntary. The participation rate was higher for the first two 
groups (100 percent for the first group, nearly 60 percent for the second group), but it 
exceeded 50 percent for the Alaska group as well. 

The 12 dimensions the students were asked to respond to were: 

1 . Satisfaction with online instruction 

2. Preference of online instruction to on-ground instruction 

3. Preference of on-ground instruction to online instruction 

4. Satisfaction with my level of learning in online courses 

5. Satisfaction with lectures in online classes 

6. Satisfaction with quality of instruction in online classes 

7. Enjoyment of threaded discussions in online classes 

8. Equity of rigor in online class to an on-ground class 

9. Equity in quality of instruction in online/on-ground classes 

10. Ability to get more personal attention in online classes 

1 1 . Online classes providing greater flexibility of time 

12. Intent to enroll in additional online classes 

Summary charts showing, in graph form, the mean response of students from the 
different groups follow. The charts are presented in clusters of similar content items with 
a brief discussion following each chart. 
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The responses in this set of graphs address the issues in the survey in inverse order. As 
can be seen, there is general satisfaction with online instruction for all three groups, 
although the first group shows a much higher degree of approval than do the latter two 
groups. Why this is so is not easy to determine, but it may be attributable to the fact that 
the first group was taught by full-time faculty with specific expertise in the content area 
in the courses. They also wrote the courses. As the program grew, a number of courses in 
the latter two groups were taught by adjunct faculty who used the materials developed by 
the full-time faculty but did not actually write the courses themselves. This may or may 
not have influenced how the students responded to the survey item regarding their 
satisfaction, but it is worth considering. 

When the students were asked about their preference for online vs. on-ground 
education — and the reverse statement of the issue — it is fairly clear that there is a decided 
preference for online instruction. This is not surprising since the students have self- 
selected to take an online program Again, though, the response is stronger for the first 
group of students who took the program. Also, it is clear that there may be slightly less 
enthusiasm among the students in Alaska, although their approval is still better than 3.00 
on a 5 point scale. Some students from Alaska may very well find themselves in the 
situation that, all things considered, they believe themselves to be better suited to be on- 
ground learners but geography makes that impossible. A close inspection of individual 
survey responses shows that eight of the Alaskan students rated the program highly (4 or 
5), but three did not (rating it 1). 
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When one looks at the expression of satisfaction with their own learning in online 
instruction, students from each of the groups rate it relatively high. In this case, the 
students from Alaska, despite lower numbers in the last graph, rate their satisfaction with 
their learning at 4. 1 8, not as high as students in group 1 at 4.71 but higher than the 
students in group 2 at 3.66. Students in group 2 were somewhat more ambivalent about 
their learning but still positive. Again the observation made above about who was 
teaching the students may influence the results. The original faculty who worked with 
group one was relatively the same as the faculty who worked with the Alaska group but 
that was not the case with group 2. 

On the other hand, the results on the other three issues — satisfaction with lectures, and 
threaded discussions (the primary modes of instruction in the programs) — showed 
approval for all three groups, with the strongest affirmation coming from group 1 
regarding the lectures in the courses.. When they were asked to rate their satisfaction with 
the quality of online instruction, the ratings followed a more predicable pattern with 
group I being the highest, group 2 next, and Alaska last. Still, the differences were 
relatively slight. 
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When one compares the equity in rigor and quality of the online and on-ground, the 
results are a bit more diverse. Students in group 1 and students from the Alaska group 
perceived the rigor to be about the same for online and on-ground instruction while 
students in group 2 were less ready to make that claim. That could be because of the 
faculty issues noted above. It could also be related to the time when the survey was 
conducted. Students in group 1 completed the survey when they had finished at least 50 
percent of their work online and the same was true for the Alaska students. The majority 
of the students in group 2 were near the beginning of their program and may have been 
unsure regarding rigor. Still, they considered online instruction to be comparable to on- 
ground instruction in rigor. Students in the other two groups tended to see the online 
program as actually more rigorous as their higher survey ratings indicated. 

There is more ambivalence regarding the equity in quality of instruction, but equity is 
still perceived to be the case, but not by as strong a rating as seen in response to other 
questions on the survey. Perhaps the best way to interpret this response is to say the 
students view the quality to be about the same for online and on-ground instruction, but 
not with as much enthusiasm as seen in other responses. 



O 



10 



9 



Intent to enroll in 
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In the first offering of the program, students in group 1 perceived that they received a 
high degree of personal attention from instructors, in fact, received more personal 
attention than in on-ground classes. This was not the case in group 2 and for the Alaska 
students. Again, the use of instructors who had not written the courses— and who 
occasionally were not as skilled in teaching online— might have been a factor for the 
group 2 students, but that was not the case with the Alaska students. The lower number 
from the Alaska students was definitely influenced by the generally lower ratings given 
by three of the eleven respondents, but they may also have expected more in terms of 
quick response than was possible for all instructors. 

Not surprisingly, students in all of the groups were quite enthusiastic about the flexibility 
of time that online instruction provides them. For many, that was the major reason they 
pursued their studies in this manner. And most of the students from all three groups, 
despite their other perceptions, plan to take additional online classes. 

On all of these twelve issues it would appear that the students, in general, with some 
reservations, are pleased with online instruction and find it to be comparable to on- 
ground instruction in providing them with learning opportunities 
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The means reported above, however, do not really tell the whole story. Comments 
provided by the students give context to their statement of attitudes. A representative 
sample of comments — taken from group two and the Alaska students — follow. 

Comments from Group Two Students 

“. . .The program is more professional than I had expected.” 

“...the professor has been flexible and the classmates super helpful.” 

“It can be frustrating when, in threaded discussions, a wide rang and number of questions 
are asked, making the requested complete paragraph response impossible. This requires a 
great deal of time to answer some of the questions....” 

“I find it is not an effective learning situation for me.” 

“This is the future. Thank you.” 

“...The workload is comparable to regular classes, but the flexibility is awesome.” 

“The online class allows me to work a t my own pace and to delve more deeply into the 
subject matter. It is definitely more challenging, but I am paying to be educated and I 
want to be challenged.” 

“...I don’t think we get to know our real capabilities.” 

“. . .Time limits should not exist. We as professionals should be able to organize our own 
scheduling....” 



Comments from Alaska Students 

“I have really enjoyed half of my classes. Those enjoyed get a ‘5’ — the others get a ‘1.’ 

Input from the prof is the key... is rigorous (maybe over the top) but I learned the 

most from his classes.... ” 

“The administrative certification program I am currently enrolled in has been very 
informative and convenient. I feel qualified to pursue my career as a school 
administrator.” 

“NU is doing a great job at establishing their online programs. I miss the personal 
interaction that comes with on-ground courses. If given the choice I would prefer a 
hybrid approach. Some courses, I believe, would be served better in “on-ground.” 

“...While the courses have been rigorous and challenging, the lack of dialog and 
interaction with professors has flattened the impact it might otherwise have....” 
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These comments reflect a variety of sentiments and mirror the results of the survey. 
Again, in general, they are positive, but not without their statement of concerns. 
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Comparison of Student Attitudes — On-ground and Online 

As interesting and helpful as it is to have a profile of student attitudes regarding their 
experiences solely from the perspective of online education, it is also useful to see how 
attitudes of online students compare to students taking the same courses at the same time 
on-ground. National University, as do most universities, invites students to complete 
course evaluations at the conclusion of each course. While the evaluation forms for both 
on-ground and online students are quite similar, there are some differences due to the 
format of instruction. Nine of the items surveyed, however, are identical and address the 
students’ attitudes regarding the quality of their learning. Those questions, which are 
responded to using a 1 — 5 point rating scale with 1 being “strongly disagree” and 5 being 
“strongly agree” are: 



1 . My writing skills have improved. 

2. My speaking skills have improved. 

3. My computer skills have improved 

4. I gained significant knowledge and skills in this subject. 

5. My skills as an independent learner have improved. 

6. I learned principles, theories, or generalizations. 

7. My ability to do research in this field has improved. 

8. I improved my critical thinking, problem-solving, or decision- 
making 

9. I can tetter apply what I have learned within diverse 
populations and situations. 

Graphs indicating the differences in responses of on-ground and online students follow. 
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Analysis Title: 


My writing skills have improved. 






Source of 
Variation 


Sum of 
Squares 


df 


Mean 

Square 


F 


Between 


0.266 


1 


0.266 


1.892 


Within 


8.871 


63 


0.141 




Total 


9.138 


64 







While some differences are noted between on-ground v. online course responses, the 
differences are not statistically significant. Students do have to write more in online 
courses, but there is no more formal writing in online classes than in on-ground classes. 
The absence of feedback on hard (no written comments in the text itself) may account for 
this slight difference. 
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Analysis Title: 


My speaking skills have improved. 




Source of 


Sum of 


Mean 




Variation 


Squares 


df Square 


F 


Between 


0.691 


1 0.691 


3.984 


Within 


10.922 


63 0.173 




Total 


11.613 


64 




The differences in on-ground v. online responses barely misses significance at .05 level 
(4.00). This would be expected since online students are limited to threaded discussion 


(written) responses and do not really use 


speaking skills at all. 





Analysis Title: 


My computer skills have improved. 






Source of 
Variation 


Sum of 
Squares 


df 


Mean 

Square 


F 


Between 


0.473 


1 


0.473 


3.793 


Within 


7.860 


63 


0.125 




Total 


8.333 


64 







The differences in responses from on-ground v. online courses closely approach 
significance at the .05 level (4.00). Again, this is to be expected in that online students are 
dependent on their computer skills to complete the course whereas on-ground students 
may or may not rely on these skills. 
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I gained significant knowledge 
and skills in this subject 



My skills as an independent 
learner have improved. 



I learned principles, theories, 
or generalizations. 



My ability to do research in 
this field has improved. 





4.16 
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X 


X 


r 

4.20 


r 

4.25 


r 

4.30 


1 

4.35 



□ Online 
M Onground 



Mean Responses 



Analysis Title: I gained significant knowledge and skills in this subject. 



Source of 
Variation 


Sum of 
Squares 


df 


Mean 

Square 


F 


Between 


0.011 


1 


0.011 


0.057 


Within 


11.914 


63 


0.189 




Total 


11.925 


64 







While some differences are noted between on-ground v. online course responses, the 
differences are small and not statistically significant. This tends to support the belief that 
there is comparable learning taking place in online and on-ground classes. 
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Analysis Title: 


My skills as an independent learner have improved. 


Source of 


Sum of 




Mean 




Variation 


Squares 


df 


Square 


F 


Between 


0.036 


1 


0.036 


0.261 


Within 


8.798 


63 


0.140 





Total 8.835 64 



The slight differences noted between on-ground and online course responses are not 
statistically significant. One would expect, as is the case here, that on-line students would 
show a slightly higher perception of their independent learning skills which are more 
essential in an online environment. 


Analysis Title: 


1 learned principles, theories, or generalizations. 


Source of 


Sum of 


Mean 


Variation 


Squares 


df Square F 


Between 


0.025 


1 0.025 0.195 


Within 


8.228 


63 0.131 


Total 


8.254 


64 


Again, slight differences are noted between on-ground and online course responses; 


however, they are not statistically significant. 


Analysis Title: 


My ability to do research in this field has improved. 


Source of 


Sum of 


Mean 


Variation 


Squares 


df Square F 


Between 


0.238 


1 0.238 1.967 


Within 


7.625 


63 0.121 


Total 


7.863 


64 



The slight differences noted between on-ground and online course responses are not 
statistically significant. 
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I improved my critical 
thinking, problem -solving, or 
decision -making. 



I can better apply what I have 
learned within diverse 
populations and situations. 




□ Online 
BOnground 
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Mean Responses 



Analysis Title: I improved my critical thinking, problem-solving, or 

decision-making. 



Source of 
Variation 


Sum of 
Squares 


df 


Mean 

Square 


F 


Between 


1.294 


1 


1.294 


8.490 


Within 


9.602 


63 


0.152 




Total 


10.895 


64 







The differences noted in on-ground v. online responses are statistically significant at the 
.005 level (8.49). Why this is so is open to speculation. It may be that on this dimension 
students miss the interaction found in the on-ground setting where it is easier to challenge 
each other and to get relatively instantaneous response. 
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Analysis Title: 


1 can better apply what 1 have learned within diverse 
populations and situations. 


Source of 


Sum of 




Mean 




Variation 


Squares 


df 


Square 


F 


Between 


0.700 


1 


0.700 


4.109 


Within 


10.728 


63 


0.170 




Total 


11.428 


64 







The differences noted in on-ground v. online responses are statistically significant at the 
.05 level (4.00). Perhaps the anonymity of the online environment is an impediment to 
appreciating diversity and does not lend itself to interacting with people from different 
backgrounds as easily as it might. This is an issue that merits further study. 



Overall, the responses to these questions, from online and a comparable sample of on- 
ground students, except for two instances, show no significant differences in the 
perceptions of the two groups. It would appear that these sets of data suggest that, for the 
most part, students rate their learning and acquisition of skills as about the same. 

Online Student Performance on Comprehensive Exit Examination 

Another indicator of how well students perform academically in the online educational 
administration program can be found by examining their scores on a comprehensive exit 
examination required of all students at the conclusion of the program. The examination is 
scored using a 5 point rubric and addresses, through essay questions, seven areas of 
content: 



1. School Leadership 

2. School Law 

3. School Personnel Administration 

4. Supervision of Instruction 

5. School Finance 

6. School-Community Relations 

7. School Restructuring 

Data on on-ground student performance in each of the content areas as well as on the 
examination as a whole are available for the past several years. Data from online 
students, since the online program is relatively new, are not as readily available. What is 
available, however, indicates that student performance is relatively comparable. A 
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comparison, keeping in mind that the “n” for online students is quite low at this time, is 
illustrated in the following chart. 
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Mean Student Score for Program, Statewide — 2000-2001 
N=125 M=4.22 

Mean Student Score for Program, Online — 2000-2001 
N=ll M=3.88 



Clearly the number of online results is too low to allow for any statistical inferences to be 
made. Nonetheless, these preliminary results do suggest that on this outcome measure, 
online students are performing at a level comparable to their fellow students who have 
completed the program on-ground. As more students complete their preliminary 
administrative services credential programs online, either totally or in part, more data will 
be available for substantive analysis. 



Student Assessment of Their Learning 

At the end of each student evaluation of a course, students are asked to rate — on a 5 point 
scale — their instructor, their assessment of their own learning, and the quality of 
instruction they have just received. A “1” represents a low evaluation and a”5” represents 
a high evaluation. Ratings for each of these three dimensions, for each of the online 
educational administration courses offered since the beginning of the online program in 
February, 2000, through the courses offered in January, 2002, are compared to ratings 
given to identical on-ground courses offered at the same time. These comparisons with 
an examination of differences are presented in the following charts. 




20 



19 






Instructor 


Learning 

Assessment Questions 


Instruction 


Analysis Title: 


Assessment of Instructor 




Source of 


Sum of 




Mean 


Variation 


Squares 


df 


Square F 


Between 


1.100 


1 


1.100 2.943 


Within 


23.535 


63 


0.374 


Total 


24.635 


64 





While some differences are noted between the responses of online and on-ground 
students, they are not statistically significant (P.05 = 4.00). It is not surprising, however, 
that the rating of the instructor would be somewhat higher on-ground since there is the 
factor of personal presence to consider. Personality and the lack of anonymity might very 
well motivate students to be a bit more generous in their response to this measure. 
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Analysis Title: 


Self Assessment of Learning 






Source of 


Sum of 




Mean 




Variation 


Squares 


df 


Square 


F 


Between 


0.108 


1 


0.108 


0.948 


Within 


7.203 


63 


0.114 




Total 


7.312 


64 







Online and on-ground students differed in their self assessment of learning; however, the 
differences again were not statistically significant. Again, the higher rating is given by 
the on-ground students and the lack of “real time” interaction might account for this 
difference in perception. Still. It must be remembered that the ratings are relatively 
similar. 



Analysis Title: 


Assessment of Instruction 






Source of 


Sum of 




Mean 




Variation 


Squares 


df 


Square 


F 


Between 


1.258 


1 


1.258 


5.063 


Within 


15.648 


63 


0.248 




Total 


16.905 


64 







The differences in online and on-ground students’ assessment of course instruction was 
found to be statistically significant at the .05 level (4.00). This dimension might be the 
most critical in this portion of the study. While, in general, students in online and on- 
ground classes rate their learning and class experiences quite similarly, online instruction, 
at this time, is somewhat limited in pedagogical strategies. It is much harder but not 
impossible, for example, to do group work and the use of videos and role-playing 
simulations presents quite a challenge in the online environment. Advances in technology 
should rectify this situation this coming year and the addition of a greater variety of 
instructional strategies might very well allow for the perceptions of both on-ground and 
online students to converge. 



Overall, there appears to be a high degree of comparability in these ratings between on- 
ground and online students. Still, it must be noted that online students tended to rate their 
experiences on these dimensions slightly lower than did the on-ground students. One 
would anticipate that as technology improves and as ways are found to overcome some of 
the anonymity issues in the online environment, the ratings would be virtually the same. 
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Also, it is appropriate to observe that not all students who are taking courses online are 
convinced that online instruction is really the best way to learn. As one student 
commented, “I find it (online instruction) is not an effective learning situation for me.” 



Instructor Perception of Student Learning 

Numbers alone and the perceptions of students alone do not give one a clear view of 
student performance in an online learning environment. The instructors of the online 
courses, all of whom have experience in teaching courses on-ground, have valuable 
insights regarding student learning and the quality of online instruction. To gain insight 
into the perceptions of the online educational administration program instructors, an 
instructor survey was completed in January, 2002. The survey addressed nine issues: 

1 . Number of online courses taught at National University 

2. Number of online courses taught at other universities 

3. Number of on-ground courses taught at National University 

4. Equity of rigor of online and on-ground courses at National University 

5. Equity of rigor of online courses taught at National University to online 
courses taught at other universities 

6. Comparability of student performance in online courses to on-ground courses 
at National University 

7. Level of student performance in online courses in relationship to participation 
in on-ground courses at National University 

8. Online student mastery of administrative competencies in relationship to on- 
ground student mastery of administrative competencies at National University 

9. General Comments 

While the response rate is low and the actual “n” is low — reflecting the fact that a number 
of instructors have taught more than one course in the program — , the insights provided 
by the instructors are helpful. Their responses follow. 

Summaiy of Responses 

1 . Number of online courses taught at National University 

N= 9 M=3.66 

(The range was 1 — 8.) 

2. Number of online courses taught at other universities 



N=9 M=NA 

(Only one respondent has taught online at another university.) 
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3. Number of on-ground classes taught at National University 

N=9 M=37.3 

(All have taught 1 -5 classes; several have taught more than 80.) 

4. Equity of rigor in online courses and on-ground courses at National University 

Online: (N=9) More Rigorous — 4; About the Same — 3; Less Rigorous — 2. 

(One instructor who rated the online course more rigorous said it was a “real 
killer.” One who said it was less rigorous expressed concern that there were 
not enough options for responding to papers, etc.” ) 

5. Equity of rigor in online courses taught at National University to online courses 
taught at other universities 

Online: The one instructor who had taught online courses at another university 
said that the courses at National University were as rigorous. 

6. Comparability of student performance in online courses to on-ground courses at 
National University 

Online: (N=9) Higher Performance — 3; Comparable Performance — 6; Lower — 0. 

7. Level of student participation in online courses in relationship to participation in 
on-ground classes at National University 

Online: (N=9) Higher Participation — 6; Same Participation — 3; Lower — 0. 

8. Online mastery of administrative competencies in relationship to on-ground 
student mastery of administrative competencies at National University 

Online: (N=9) Higher Mastery — 0; About the Same — 7; Lower — 2. 

(The two instructors who rated student mastery of competencies as lower 
expressed concern that there was a need for more practical, in-basket types of 
activities as well as a greater need to get to know the students personally before 
making a judgment on this issue.) 



9. Comments 

The comments of the instructors follow: 

“...very well organized. If anything, extremely heavy workload for students.” 
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“The program is strong, but there is always room for improvement. Some students 
occasionally mimic the response of their classmates in their threaded discussion 
answers.” 

“Academic quality is high online. Students are mature, experienced educators 
online that typically want to gain skills to improve their school site performance.” 

“I think it is a very viable method of instruction — as instructors learn more about 
how to best structure the classes — and the university does a good job of reviewing 
the syllabi that have been written and keeping them current — I think the quality of 
instruction will improve. I do strongly feel that no student should be able to earn 
the entire administrative degree without some on-ground component — to ensure 
that we are not dealing with a “virtual student”, but with a student who does 
indeed have the competencies necessary for school administration.” 

‘1 think it is great— as we become more able in design, we can really enrich the 
online classes far beyond in-class instruction.” 

“I look forward to a continued growth and development in the online classes and 
that is exciting.” 

“Because students work at their own pace there is little interaction among 
students. They accomplish the assignments but don’t discuss much among 
themselves.” 

“The students found the course to be a quite a challenge. They commented on 
how much work was involved in this course. They enjoy their online 
communication, especially since many of them are far away from us in San 
Diego. They also enjoyed interacting with educators from all levels of education 
including preschool, primary, secondary, adult, and the Department of 
Corrections. What they did find out was that good quality supervision models can 
be applied at all levels of education.” 

“After spending quality time communicating with the students and requiring 
adherence to APA format and quality written work, I felt that the students were 
generally of high caliber. There were some exceptions as in every class, but the 
responses to threaded discussions were very thorough. It is my belief that the key 
to academic success is the degree to which the instructor requires timely and 
professional contributions.” 

The above comments as well as the other responses to the instructor, with a few caveats, 
strongly support the online educational administration program being offered at National 
University. There is strong consensus that the online courses are quite comparable in 
quality and rigor to on-ground courses and that students are receiving the instruction 
they need to become successful school administrators. What remains a challenge is the 
“personalization” of online classes, particularly getting to know the students and 
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providing them with a greater array of practical, hand-on activities that will enrich their 
on-the-job problem solving skills. This is an issue faculty must address as they continue 
to work in the online environment. 



General Conclusions 

The central question of this inquiry remains, “How do we know what the online students 
have learned and is it comparable to what students have learned in on-ground classes? ” 
This is not an easy question to answer, but the information gleaned in reflecting on the 
data examined in response to the four sub-questions discussed above provides us with us 
some tentative conclusions. When one examines student attitudes toward online 
instruction in educational administration at National University in conjunction with their 
evaluation of instructors, the quality of their courses, their own learning, their 
performance on comprehensive exit examinations, and the perceptions of their 
instructors, the following findings emerge. 

1 . There is a high degree of comparability in the quality and rigor of the 
online and on-ground educational administration courses. 

2. Online students are generally satisfied with the quality of instruction 
in their program. 

3. Students, generally, academically perform comparably in online and 
on-ground educational administration courses. Both groups meet 
established standards of mastery. 

4. Online instructors are quite positive about the online educational 
administration program. 

5. Students enrolled in online courses, even if they have concerns, like the 
the flexibility it gives them and plan to take more courses in this form. 

Perhaps it is too soon to argue whether or not the central question of this inquiry has been 
answered satisfactorily. Currently, there are some very slight differences in stated student 
perceptions that some observers might see as possibly favoring on-ground instruction. 
These differences, except for a very few instances, as has been noted, are not statistically 
significant. There are many variables that need to be taken into account when analyzing 
these data, variables that may very well have a significant influence on these findings. A 
beginning, however, has been made and it strongly suggests that online instruction in 
educational administration will be and can be a significant aspect of administrator 
preparation in the future. 
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